After more than 13 years, I've launched a new design for the FAI
project web site
https://fai-project.org
It now uses Materialize CSS and will work much better on mobile
devices. Thanks to Thorsten B lo who did the first part of converting
the web pages to the new design.
I hope you all enjoy the new layout.
FAI
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
This is the first monthly report in 2023.
After more than a year, a new major FAI release is ready to download.
Following new features are included:
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
In my two most recent posts, I listed the fiction and classic fiction I enjoyed the most in 2022.
I'll leave my roundup of general non-fiction until tomorrow, but today I'll be going over my favourite memoirs and biographies, in no particular order.
Books that just missed the cut here include Roisin Kiberd's The Disconnect: A Personal Journey Through the Internet (2019), Steve Richards' The Prime Ministers (2019) which reflects on UK leadership from Harold Wilson to Boris Johnson, Robert Graves Great War memoir Goodbye to All That (1929) and David Mikics's portrait of Stanley Kubrick called American Filmmaker.
Afropean: Notes from Black Europe (2019) Johny Pitts Johny Pitts is a photographer and writer who lives in the north of England who set out to explore "black Europe from the street up" those districts within European cities that, although they were once 'white spaces' in the past, they are now occupied by Black people. Unhappy with the framing of the Black experience back home in post-industrial Sheffield, Pitts decided to become a nomad and goes abroad to seek out the sense of belonging he cannot find in post-Brexit Britain, and Afropean details his journey through Paris, Brussels, Lisbon, Berlin, Stockholm and Moscow. However, Pitts isn't just avoiding the polarisation and structural racism embedded in contemporary British life. Rather, he is seeking a kind of super-national community that transcends the reductive and limiting nationalisms of all European countries, most of which have based their national story on a self-serving mix of nostalgia and postcolonial fairy tales. Indeed, the term 'Afropean' is the key to understanding the goal of this captivating memoir. Pitts writes at the beginning of this book that the word wasn't driven only as a response to the crude nativisms of Nigel Farage and Marine Le Pen, but that it:
encouraged me to think of myself as whole and unhyphenated. [ ] Here was a space where blackness was taking part in shaping European identity at large. It suggested the possibility of living in and with more than one idea: Africa and Europe, or, by extension, the Global South and the West, without being mixed-this, half-that or black-other. That being black in Europe didn t necessarily mean being an immigrant.In search of this whole new theory of home, Pitts travels to the infamous banlieue of Clichy-sous-Bois just to the East of Paris, thence to Matong in Brussels, as well as a quick and abortive trip into Moscow and other parallel communities throughout the continent. In these disparate environs, Pitts strikes up countless conversations with regular folk in order to hear their quotidian stories of living, and ultimately to move away from the idea that Black history is defined exclusively by slavery. Indeed, to Pitts, the idea of race is one that ultimately restricts one's humanity; the concept "is often forced to embody and speak for certain ideas, despite the fact it can't ever hold in both hands the full spectrum of a human life and the cultural nuances it creates." It's difficult to do justice to the effectiveness of the conversations Pitts has throughout his travels, but his shrewd attention to demeanour, language, raiment and expression vividly brings alive the people he talks to. Of related interest to fellow Brits as well are the many astute observations and comparisons with Black and working-class British life. The tone shifts quite often throughout this book. There might be an amusing aside one minute, such as the portrait of an African American tourist in Paris to whom "the whole city was a film set, with even its homeless people appearing to him as something oddly picturesque." But the register abruptly changes when he visits Clichy-sous-Bois on an anniversary of important to the area, and an element of genuine danger is introduced when Johny briefly visits Moscow and barely gets out alive. What's especially remarkable about this book is there is a freshness to Pitt s treatment of many well-worn subjects. This can be seen in his account of Belgium under the reign of Leopold II, the history of Portuguese colonialism (actually mostly unknown to me), as well in the way Pitts' own attitude to contemporary anti-fascist movements changes throughout an Antifa march. This chapter was an especial delight, and not only because it underlined just how much of Johny's trip was an inner journey of an author willing have his mind changed. Although Johny travels alone throughout his journey, in the second half of the book, Pitts becomes increasingly accompanied by a number of Black intellectuals by the selective citing of Frantz Fanon and James Baldwin and Caryl Phillips. (Nevertheless, Jonny has also brought his camera for the journey as well, adding a personal touch to this already highly-intimate book.) I suspect that his increasing exercise of Black intellectual writing in the latter half of the book may be because Pitts' hopes about 'Afropean' existence ever becoming a reality are continually dashed and undercut. The unity among potential Afropeans appears more-and-more unrealisable as the narrative unfolds, the various reasons of which Johny explores both prosaically and poetically. Indeed, by the end of the book, it's unclear whether Johny has managed to find what he left the shores of England to find. But his mix of history, sociology and observation of other cultures right on my doorstep was something of a revelation to me.
Orwell's Roses (2021)
Rebecca Solnit
Orwell s Roses is an alternative journey through the life and afterlife of George Orwell, reimaging his life primarily through the lens of his attentiveness to nature. Yet this framing of the book as an 'alternative' history is only revisionist if we compare it to the usual view of Orwell as a bastion of 'free speech' and English 'common sense' the roses of the title of this book were very much planted by Orwell in his Hertfordshire garden in 1936, and his yearning of nature one was one of the many constants throughout his life. Indeed, Orwell wrote about wildlife and outdoor life whenever he could get away with it, taking pleasure in a blackbird's song and waxing nostalgically about the English countryside in his 1939 novel Coming Up for Air (reviewed yesterday).
By sheer chance, I actually visited this exact garden immediately to the publication of this book
The Disaster Artist (2013)
Greg Sestero & Tom Bissell
For those not already in the know, The Room was a 2003 film by director-producer-writer-actor Tommy Wiseau, an inscrutable Polish immigr with an impenetrable background, an idiosyncratic choice of wardrobe and a mysterious large source of income. The film, which centres on a melodramatic love triangle, has since been described by several commentators and publications as one of the worst films ever made.
Tommy's production completely bombed at the so-called 'box office' (the release was actually funded entirely by Wiseau personally), but the film slowly became a favourite at cult cinema screenings. Given Tommy's prominent and central role in the film, there was always an inherent cruelty involved in indulging in the spectacle of The Room the audience was laughing because the film was astonishingly bad, of course, but Wiseau infused his film with sincere earnestness that in a heartless twist of irony may be precisely why it is so terrible to begin with. Indeed, it should be stressed that The Room is not simply a 'bad' film, and therefore not worth paying any attention to: it is uncannily bad in a way that makes it eerily compelling to watch. It unintentionally subverts all the rules of filmmaking in a way that captivates the attention. Take this representative example:
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
Holger Levsen: Welcome, David, thanks for taking the time to talk with us today. First, could you briefly tell me about yourself?
David: Sure! I m David A. Wheeler and
I work for the Linux Foundation as the Director of Open Source Supply Chain Security.
That just means that my job is to help open source software projects
improve their security, including its development, build, distribution,
and incorporation in larger works, all the way out to its eventual use by end-users.
In my copious free time I also teach at George Mason University (GMU); in particular,
I teach a graduate course on how to design and implement secure software.
My background is technical. I have a Bachelor s in Electronics Engineering,
a Master s in Computer Science and a PhD in Information Technology.
My PhD dissertation is connected to reproducible builds.
My PhD dissertation was on countering the Trusting Trust attack, an attack
that subverts fundamental build system tools such as compilers.
The attack was discovered by Karger & Schell in the 1970s, and later
demonstrated & popularized by Ken Thompson.
In my dissertation on trusting trust I showed that a process
called Diverse Double-Compiling (DDC) could detect trusting trust attacks.
That process is a specialized kind of reproducible build specifically designed
to detect trusting trust style attacks. In addition, countering the trusting trust
attack primarily becomes more important only when reproducible builds become
more common. Reproducible builds enable detection of
build-time subversions.
Most attackers wouldn t bother with a trusting trust attack if they could just
directly use a build-time subversion of the software they actually want to subvert.
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
My impression of that city from couple of visits at that point in time where they were still more tongas (horse-ridden carriages), an occasional two wheelers and not many three wheelers. Although, it was one of the more turbulent times as lot of agitation for worker rights were happening around that time and a lot of industrial action. Later that led to lot of closure of manufacturing in Bombay and it became more commercial. It would be interesting to know whether they shot it in actual India or just made a set somewhere in Australia, where it possibly might have been shot. The chawl of the book needs a bit of arid land and Australia has lots of it.
It is also interesting as this was a project that had who s who interested in it for a long time but somehow none of them was able to bring the project to fruition, the project seems to largely have an Australian cast as well as second generations of Indians growing in Australia. To take names, Amitabh Bacchan, Johnny Depp, Russel Crowe each of them wanted to make it into a feature film. In retrospect, it is good it was not into a movie, otherwise they would have to cut a lot of material and that perhaps wouldn t have been sufficient. Making it into a web series made sure they could have it in multiple seasons if people like it. There is a lot between now and 12 episodes to even guess till where it would leave you then. So, if you have not read the book and have some holidays coming up, can recommend it. The writing IIRC is easy and just flows. There is a bit of action but much more nuance in the book while in the web series they are naturally more about action. There is also quite a bit of philosophy between him and Kaderbhai and while the series touches upon it, it doesn t do justice but then again it is being commercially made.
Read the book, see the series and share your thoughts on what you think. It is possible that the series might go up or down but am sharing from where I see it, may do another at the end of the season, depending on where they leave it and my impressions.
Update A slight update from the last blog post. Seems Rishi Sunak seems would be made PM of UK. With Hunt as chancellor and Rishi Sunak, Austerity 2.0 seems complete. There have been numerous articles which share how austerity gives rises to fascism and vice-versa. History gives lot of lessons about the same. In Germany, when the economy was not good, it was all blamed on the Jews for number of years. This was the reason for rise of Hitler, and while it did go up by a bit, propaganda by him and his loyalists did the rest. And we know and have read about the Holocaust. Today quite a few Germans deny it or deny parts of it but that s how misinformation spreads. Also Hitler is looked now more as an aberration rather than something to do with the German soul. I am not gonna talk more as there is still lots to share and that actually perhaps requires its own blog post to do justice for the same.
So the last three stories I found the most intriguing.
The first one is titled Man on the Beach. Apparently, a gentleman goes to one of the beaches, a sort of lonely beach, hails a taxi and while returning suddenly dies. The Taxi driver showing good presence of mind takes it to hospital where the gentleman is declared dead on arrival. Unlike in India, he doesn t run away but goes to the cafeteria and waits there for the cops to arrive and take his statement. Now the man is in his early 40s and looks to be fit. Upon searching his pockets he is found to relatively well-off and later it turns out he owns a couple of shops. So then here are the questions ?
What was the man doing on a beach, in summer that beach is somewhat popular but other times not so much, so what was he doing there?
How did he die, was it a simple heart attack or something more? If he had been drugged or something then when and how?
These and more questions can be answered by reading the story Man on the Beach .
2. The death of a photographer Apparently, Kurt lives in a small town where almost all the residents have been served one way or the other by the town photographer. The man was polite and had worked for something like 40 odd years before he is killed/murdered. Apparently, he is murdered late at night. So here come the questions
a. The shop doesn t even stock any cameras and his cash box has cash. Further investigation reveals it is approximate to his average takeout for the day. So if it s not for cash, then what is the motive ?
b. The body was discovered by his cleaning staff who has worked for almost 20 years, 3 days a week. She has her own set of keys to come and clean the office? Did she give the keys to someone, if yes why?
c. Even after investigation, there is no scandal about the man, no other woman or any vices like gambling etc. that could rack up loans. Also, nobody seems to know him and yet take him for granted till he dies. The whole thing appears to be quite strange. Again, the answers lie in the book.
3. The Pyramid Kurt is sleeping one night when the telephone rings. The scene starts with a Piper Cherokee, a single piston aircraft flying low and dropping something somewhere or getting somebody from/on the coast of Sweden. It turns and after a while crashes. Kurt is called to investigate it. Turns out, the plane was supposed to be destroyed. On crash, both the pilot and the passenger are into pieces so only dental records can prove who they are. Same day or a day or two later, two seemingly ordinary somewhat elderly women, spinsters, by all accounts, live above the shop where they sell buttons and all kinds of sewing needs of the town. They seem middle-class. Later the charred bodies of the two sisters are found :(. So here come the questions
a.Did the plane drop something or pick something somebody up ? The Cherokee is a small plane so any plane field or something it could have landed up or if a place was somehow marked then could be dropped or picked up without actually landing.
b. The firefighter suspects arson started at multiple places with the use of petrol? The question is why would somebody wanna do that? The sisters don t seem to be wealthy and practically everybody has bought stuff from them. They weren t popular but weren t also unpopular.
c. Are the two crimes connected or unconnected? If connected, then how?
d. Most important question, why the title Pyramid is given to the story. Why does the author share the name Pyramid. Does he mean the same or the original thing? He could have named it triangle. Again, answers to all the above can be found in the book.
One thing I also became very aware of during reading the book that it is difficult to understand people s behavior and what they do. And this is without even any criminality involved in. Let s say for e.g. I die in some mysterious circumstances, the possibility of the police finding my actions in last days would be limited and this is when I have hearing loss. And this probably is more to do with how our minds are wired. And most people I know are much more privacy conscious/aware than I am.
I m on the train back from MuniHac, where I gave a talk about the rec-def library that I have excessively blogged about recently (here, here, here and here). I got quite flattering comments about that talk, so if you want to see if they were sincere, I suggest you watch the recording of Getting recursive definitions off their bottoms (but it s not necessary for the following).
After the talk, Franz Thoma approached me and told me a story of how we was once implementing the game Minesweeper in Haskell, and in particular the part of the logic where, after the user has uncovered a field, the game would automatically uncover all fields that are next to a neutral field, i.e. one with zero adjacent bombs. He was using a comonadic data structure, which makes a context-dependent parallel computation such as uncovering one field quite natural, and was hoping that using a suitable fix-point operator, he can elegantly obtain not just the next step, but directly the result of recursively uncovering all these fields. But, much to his disappointment, that did not work out: Due to the recursion inherent in that definition, a knot-tying fixed-point operator will lead to a cyclic definition.
Microsoft Minesweeper
rec-def library could have helped him, and we sat down to find out, and this is the tale of this blog post. I will avoid the comonadic abstractions and program it more naively, though, to not lose too many readers along the way. Maybe read Chris Penner s blog post and Finch s functional pearl Getting a Quick Fix on Comonads if you are curious about that angle.
Array data type, it s Ix-based indexing is quite useful for grids:
The library lacks a function to generate an array from a generating function, but it is easy to add:
Let s also fix the size of the board, as a pair of lower and upper bounds (this is the format that the Ix type class needs):
Now board is simply a grid of boolean values, with True indicating that a bomb is there:
type Board = Grid Bool board1 :: Board board1 = listArray size [ False, False, False, False , True, False, False, False , True, False, False, False , False, False, False, False ]
*), we can print the board:
pGrid :: (C -> String) -> String pGrid p = unlines [ concat [ p' (y,x) x <- [lx-1 .. ux+1] ] y <- [ly-1 .. uy+1] ] where ((lx,ly),(ux,uy)) = size p' c inRange size c = p c p' _ = "#" pBombs :: Board -> String pBombs b = pGrid $ \c -> if b ! c then "*" else " "
b ! c looks up a the coordinate in the array, and is True when there is a bomb at that coordinate.
So here is our board, with two bombs:
ghci> putStrLn $ pBombs board1
######
# #
#* #
#* #
# #
######inRange:
neighbors :: C -> [C]
neighbors (x,y) =
[ c
(dx, dy) <- range ((-1,-1), (1,1))
, (dx, dy) /= (0,0)
, let c = (x + dx, y + dy)
, inRange size c
]data H = Bomb Hint Int deriving Eq hint :: Board -> C -> H hint b c b ! c = Bomb hint b c = Hint $ sum [ 1 c' <- neighbors c, b ! c' ]
hint :: Board -> C -> H hint b c b ! c = Bomb hint b c = Hint $ sum [ 1 c' <- neighbors c, b ! c' ] pCell :: Board -> C -> String pCell b c = case hint b c of Bomb -> "*" Hint 0 -> " " Hint n -> show n pBoard :: Board -> String pBoard b = pGrid (pCell b)
ghci> putStrLn $ pBoard board1
######
#11 #
#*2 #
#*2 #
#11 #
######?:
type Mask = Grid Bool mask1 :: Mask mask1 = listArray size [ True, True, True, False , False, False, False, False , False, False, False, False , False, False, False, False ] pMasked :: Board -> Mask -> String pMasked b m = pGrid $ \c -> if m ! c then pCell b c else "?"
ghci> putStrLn $ pMasked board1 mask1
######
#11 ?#
#????#
#????#
#????#
######solve0 :: Board -> Mask -> Mask solve0 b m0 = m1 where m1 :: Mask m1 = genArray size $ \c -> m0 ! c or [ m0 ! c' c' <- neighbors c, hint b c' == Hint 0 ]
m1 from the old one m0 by the following logic: A field is visible if it was visible before (m0 ! c), or if any of its neighboring, neutral fields are visible.
This works so far: I uncovered the three fields next to the one neutral visible field:
ghci> putStrLn $ pMasked board1 $ solve0 board1 mask1
######
#11 #
#?2 #
#????#
#????#
######m0 ! c), or if any of its neighboring, neutral fields will be visible.
In the code, this is just a single character change: Instead of looking at m0 to see if a neighbor is visible, we look at m1:
solve1 :: Board -> Mask -> Mask solve1 b m0 = m1 where m1 :: Mask m1 = genArray size $ \c -> m0 ! c or [ m1 ! c' c' <- neighbors c, hint b c' == Hint 0 ]
kfix comonadic fixed-point operator in his code, I believe.)
Does it work? It seems so:
ghci> putStrLn $ pMasked board1 $ solve1 board1 mask1
######
#11 #
#?2 #
#?2 #
#?1 #
######mask2 :: Mask
mask2 = listArray size
[ True, True, False, False
, False, False, False, False
, False, False, False, False
, False, False, False, True
]ghci> putStrLn $ pMasked board1 mask2
######
#11??#
#????#
#????#
#??? #
######solve1 function does not work, and just sits there:
ghci> putStrLn $ pMasked board1 $ solve1 board1 mask2
######
#11^CInterrupted. and or), this does not make progress.
It worked with mask1 more or less by accident: None of the fields on in the first column don t have neutral neighbors, so nothing happens there. And for all the fields in the third and forth column, the code will know for sure that they will be uncovered based on their upper neighbors, which come first in the neighbors list, and due to the short-circuting properties of , it doesn t have to look at the later cells, and the vicious cycle is avoided.
rec-def comes in: By using the RBool type in m1 instead of plain Bool, the recursive self-reference is not a problem, and it simply works:
import qualified Data.Recursive.Bool as RB solve2 :: Board -> Mask -> Mask solve2 b m0 = fmap RB.get m1 where m1 :: Grid RB.RBool m1 = genArray size $ \c -> RB.mk (m0 ! c) RB. RB.or [ m1 ! c' c' <- neighbors c, hint b c' == Hint 0 ]
m1; I just replaced Bool with RBool, with RB. and or with RB.or. And used RB.get at the end to get a normal boolean out. And , here we go:
ghci> putStrLn $ pMasked board1 $ solve2 board1 mask2
######
#11 #
#?2 #
#?2 #
#?1 #
######rec-def helps , which always end up a bit anti-climatic because it just works , at least in these cases. Hope you enjoyed it nevertheless.
This post describes how I ve put together a simple static content server for
kubernetes clusters using a Pod with a persistent volume and multiple
containers: an sftp server to manage contents, a web server to publish them
with optional access control and another one to run scripts which need access
to the volume filesystem.
The sftp server runs using
MySecureShell, the web
server is nginx and the script runner uses the
webhook tool to publish endpoints to call
them (the calls will come from other Pods that run backend servers or are
executed from Jobs or CronJobs).
NodeJS API with endpoints to upload
files and store them on S3 compatible services that were later accessed via
HTTPS, but the requirements changed and we needed to be able to publish folders
instead of individual files using their original names and apply access
restrictions using our API.
Thinking about our requirements the use of a regular filesystem to keep the
files and folders was a good option, as uploading and serving files is simple.
For the upload I decided to use the sftp protocol, mainly because I already
had an sftp container image based on
mysecureshell prepared; once
we settled on that we added sftp support to the API server and configured it
to upload the files to our server instead of using S3 buckets.
To publish the files we added a nginx container configured
to work as a reverse proxy that uses the
ngx_http_auth_request_module
to validate access to the files (the sub request is configurable, in our
deployment we have configured it to call our API to check if the user can
access a given URL).
Finally we added a third container when we needed to execute some tasks
directly on the filesystem (using kubectl exec with the existing containers
did not seem a good idea, as that is not supported by CronJobs objects, for
example).
The solution we found avoiding the NIH Syndrome (i.e. write our own tool) was
to use the webhook tool to provide the
endpoints to call the scripts; for now we have three:
PATH,hardlink all the files that are identical on the filesystem,mysecureshell container can be used to provide an sftp service with
multiple users (although the files are owned by the same UID and GID) using
standalone containers (launched with docker or podman) or in an
orchestration system like kubernetes, as we are going to do here.
The image is generated using the following Dockerfile:
ARG ALPINE_VERSION=3.16.2
FROM alpine:$ALPINE_VERSION as builder
LABEL maintainer="Sergio Talens-Oliag <sto@mixinet.net>"
RUN apk update &&\
apk add --no-cache alpine-sdk git musl-dev &&\
git clone https://github.com/sto/mysecureshell.git &&\
cd mysecureshell &&\
./configure --prefix=/usr --sysconfdir=/etc --mandir=/usr/share/man\
--localstatedir=/var --with-shutfile=/var/lib/misc/sftp.shut --with-debug=2 &&\
make all && make install &&\
rm -rf /var/cache/apk/*
FROM alpine:$ALPINE_VERSION
LABEL maintainer="Sergio Talens-Oliag <sto@mixinet.net>"
COPY --from=builder /usr/bin/mysecureshell /usr/bin/mysecureshell
COPY --from=builder /usr/bin/sftp-* /usr/bin/
RUN apk update &&\
apk add --no-cache openssh shadow pwgen &&\
sed -i -e "s ^.*\(AuthorizedKeysFile\).*$ \1 /etc/ssh/auth_keys/%u "\
/etc/ssh/sshd_config &&\
mkdir /etc/ssh/auth_keys &&\
cat /dev/null > /etc/motd &&\
add-shell '/usr/bin/mysecureshell' &&\
rm -rf /var/cache/apk/*
COPY bin/* /usr/local/bin/
COPY etc/sftp_config /etc/ssh/
COPY entrypoint.sh /
EXPOSE 22
VOLUME /sftp
ENTRYPOINT ["/entrypoint.sh"]
CMD ["server"]/etc/sftp_config file is used to
configure
the mysecureshell server to have all the user homes under /sftp/data, only
allow them to see the files under their home directories as if it were at the
root of the server and close idle connections after 5m of inactivity:
# Default mysecureshell configuration
<Default>
# All users will have access their home directory under /sftp/data
Home /sftp/data/$USER
# Log to a file inside /sftp/logs/ (only works when the directory exists)
LogFile /sftp/logs/mysecureshell.log
# Force users to stay in their home directory
StayAtHome true
# Hide Home PATH, it will be shown as /
VirtualChroot true
# Hide real file/directory owner (just change displayed permissions)
DirFakeUser true
# Hide real file/directory group (just change displayed permissions)
DirFakeGroup true
# We do not want users to keep forever their idle connection
IdleTimeOut 5m
</Default>
# vim: ts=2:sw=2:etentrypoint.sh script is the one responsible to prepare the container for
the users included on the /secrets/user_pass.txt file (creates the users with
their HOME directories under /sftp/data and a /bin/false shell and
creates the key files from /secrets/user_keys.txt if available).
The script expects a couple of environment variables:
SFTP_UID: UID used to run the daemon and for all the files, it has to be
different than 0 (all the files managed by this daemon are going to be
owned by the same user and group, even if the remote users are different).SFTP_GID: GID used to run the daemon and for all the files, it has to be
different than 0.SSH_PORT and SSH_PARAMS values if present.
It also requires the following files (they can be mounted as secrets in
kubernetes):
/secrets/host_keys.txt: Text file containing the ssh server keys in mime
format; the file is processed using the reformime utility (the one included
on busybox) and can be generated using the
gen-host-keys script included on the container (it uses ssh-keygen and
makemime)./secrets/user_pass.txt: Text file containing lines of the form
username:password_in_clear_text (only the users included on this file are
available on the sftp server, in fact in our deployment we use only the
scs user for everything)./secrets/user_keys.txt: Text file that contains lines of the form
username:public_ssh_ed25519_or_rsa_key; the public keys are installed on
the server and can be used to log into the sftp server if the username
exists on the user_pass.txt file.entrypoint.sh script are:
#!/bin/sh
set -e
# ---------
# VARIABLES
# ---------
# Expects SSH_UID & SSH_GID on the environment and uses the value of the
# SSH_PORT & SSH_PARAMS variables if present
# SSH_PARAMS
SSH_PARAMS="-D -e -p $ SSH_PORT:=22 $ SSH_PARAMS "
# Fixed values
# DIRECTORIES
HOME_DIR="/sftp/data"
CONF_FILES_DIR="/secrets"
AUTH_KEYS_PATH="/etc/ssh/auth_keys"
# FILES
HOST_KEYS="$CONF_FILES_DIR/host_keys.txt"
USER_KEYS="$CONF_FILES_DIR/user_keys.txt"
USER_PASS="$CONF_FILES_DIR/user_pass.txt"
USER_SHELL_CMD="/usr/bin/mysecureshell"
# TYPES
HOST_KEY_TYPES="dsa ecdsa ed25519 rsa"
# ---------
# FUNCTIONS
# ---------
# Validate HOST_KEYS, USER_PASS, SFTP_UID and SFTP_GID
_check_environment()
# Check the ssh server keys ... we don't boot if we don't have them
if [ ! -f "$HOST_KEYS" ]; then
cat <<EOF
We need the host keys on the '$HOST_KEYS' file to proceed.
Call the 'gen-host-keys' script to create and export them on a mime file.
EOF
exit 1
fi
# Check that we have users ... if we don't we can't continue
if [ ! -f "$USER_PASS" ]; then
cat <<EOF
We need at least the '$USER_PASS' file to provision users.
Call the 'gen-users-tar' script to create a tar file to create an archive that
contains public and private keys for users, a 'user_keys.txt' with the public
keys of the users and a 'user_pass.txt' file with random passwords for them
(pass the list of usernames to it).
EOF
exit 1
fi
# Check SFTP_UID
if [ -z "$SFTP_UID" ]; then
echo "The 'SFTP_UID' can't be empty, pass a 'GID'."
exit 1
fi
if [ "$SFTP_UID" -eq "0" ]; then
echo "The 'SFTP_UID' can't be 0, use a different 'UID'"
exit 1
fi
# Check SFTP_GID
if [ -z "$SFTP_GID" ]; then
echo "The 'SFTP_GID' can't be empty, pass a 'GID'."
exit 1
fi
if [ "$SFTP_GID" -eq "0" ]; then
echo "The 'SFTP_GID' can't be 0, use a different 'GID'"
exit 1
fi
# Adjust ssh host keys
_setup_host_keys()
opwd="$(pwd)"
tmpdir="$(mktemp -d)"
cd "$tmpdir"
ret="0"
reformime <"$HOST_KEYS" ret="1"
for kt in $HOST_KEY_TYPES; do
key="ssh_host_$ kt _key"
pub="ssh_host_$ kt _key.pub"
if [ ! -f "$key" ]; then
echo "Missing '$key' file"
ret="1"
fi
if [ ! -f "$pub" ]; then
echo "Missing '$pub' file"
ret="1"
fi
if [ "$ret" -ne "0" ]; then
continue
fi
cat "$key" >"/etc/ssh/$key"
chmod 0600 "/etc/ssh/$key"
chown root:root "/etc/ssh/$key"
cat "$pub" >"/etc/ssh/$pub"
chmod 0600 "/etc/ssh/$pub"
chown root:root "/etc/ssh/$pub"
done
cd "$opwd"
rm -rf "$tmpdir"
return "$ret"
# Create users
_setup_user_pass()
opwd="$(pwd)"
tmpdir="$(mktemp -d)"
cd "$tmpdir"
ret="0"
[ -d "$HOME_DIR" ] mkdir "$HOME_DIR"
# Make sure the data dir can be managed by the sftp user
chown "$SFTP_UID:$SFTP_GID" "$HOME_DIR"
# Allow the user (and root) to create directories inside the $HOME_DIR, if
# we don't allow it the directory creation fails on EFS (AWS)
chmod 0755 "$HOME_DIR"
# Create users
echo "sftp:sftp:$SFTP_UID:$SFTP_GID:::/bin/false" >"newusers.txt"
sed -n "/^[^#]/ s/:/ /p " "$USER_PASS" while read -r _u _p; do
echo "$_u:$_p:$SFTP_UID:$SFTP_GID::$HOME_DIR/$_u:$USER_SHELL_CMD"
done >>"newusers.txt"
newusers --badnames newusers.txt
# Disable write permission on the directory to forbid remote sftp users to
# remove their own root dir (they have already done it); we adjust that
# here to avoid issues with EFS (see before)
chmod 0555 "$HOME_DIR"
# Clean up the tmpdir
cd "$opwd"
rm -rf "$tmpdir"
return "$ret"
# Adjust user keys
_setup_user_keys()
if [ -f "$USER_KEYS" ]; then
sed -n "/^[^#]/ s/:/ /p " "$USER_KEYS" while read -r _u _k; do
echo "$_k" >>"$AUTH_KEYS_PATH/$_u"
done
fi
# Main function
exec_sshd()
_check_environment
_setup_host_keys
_setup_user_pass
_setup_user_keys
echo "Running: /usr/sbin/sshd $SSH_PARAMS"
# shellcheck disable=SC2086
exec /usr/sbin/sshd -D $SSH_PARAMS
# ----
# MAIN
# ----
case "$1" in
"server") exec_sshd ;;
*) exec "$@" ;;
esac
# vim: ts=2:sw=2:ethost_keys.txt file as follows:
$ docker run --rm stodh/mysecureshell gen-host-keys > host_keys.txt#!/bin/sh
set -e
# Generate new host keys
ssh-keygen -A >/dev/null
# Replace hostname
sed -i -e 's/@.*$/@mysecureshell/' /etc/ssh/ssh_host_*_key.pub
# Print in mime format (stdout)
makemime /etc/ssh/ssh_host_*
# vim: ts=2:sw=2:et.tar file that contains auth data
for the list of usernames passed to it (the file contains a user_pass.txt
file with random passwords for the users, public and private ssh keys for them
and the user_keys.txt file that matches the generated keys).
To generate a tar file for the user scs we can execute the following:
$ docker run --rm stodh/mysecureshell gen-users-tar scs > /tmp/scs-users.taruser_pass.txt file we can do:
$ tar tvf /tmp/scs-users.tar
-rw-r--r-- root/root 21 2022-09-11 15:55 user_pass.txt
-rw-r--r-- root/root 822 2022-09-11 15:55 user_keys.txt
-rw------- root/root 387 2022-09-11 15:55 id_ed25519-scs
-rw-r--r-- root/root 85 2022-09-11 15:55 id_ed25519-scs.pub
-rw------- root/root 3357 2022-09-11 15:55 id_rsa-scs
-rw------- root/root 3243 2022-09-11 15:55 id_rsa-scs.pem
-rw-r--r-- root/root 729 2022-09-11 15:55 id_rsa-scs.pub
$ tar xfO /tmp/scs-users.tar user_pass.txt
scs:20JertRSX2Eaar4x#!/bin/sh
set -e
# ---------
# VARIABLES
# ---------
USER_KEYS_FILE="user_keys.txt"
USER_PASS_FILE="user_pass.txt"
# ---------
# MAIN CODE
# ---------
# Generate user passwords and keys, return 1 if no username is received
if [ "$#" -eq "0" ]; then
return 1
fi
opwd="$(pwd)"
tmpdir="$(mktemp -d)"
cd "$tmpdir"
for u in "$@"; do
ssh-keygen -q -a 100 -t ed25519 -f "id_ed25519-$u" -C "$u" -N ""
ssh-keygen -q -a 100 -b 4096 -t rsa -f "id_rsa-$u" -C "$u" -N ""
# Legacy RSA private key format
cp -a "id_rsa-$u" "id_rsa-$u.pem"
ssh-keygen -q -p -m pem -f "id_rsa-$u.pem" -N "" -P "" >/dev/null
chmod 0600 "id_rsa-$u.pem"
echo "$u:$(pwgen -s 16 1)" >>"$USER_PASS_FILE"
echo "$u:$(cat "id_ed25519-$u.pub")" >>"$USER_KEYS_FILE"
echo "$u:$(cat "id_rsa-$u.pub")" >>"$USER_KEYS_FILE"
done
tar cf - "$USER_PASS_FILE" "$USER_KEYS_FILE" id_* 2>/dev/null
cd "$opwd"
rm -rf "$tmpdir"
# vim: ts=2:sw=2:etnginx-scs container is generated using the following Dockerfile:
ARG NGINX_VERSION=1.23.1
FROM nginx:$NGINX_VERSION
LABEL maintainer="Sergio Talens-Oliag <sto@mixinet.net>"
RUN rm -f /docker-entrypoint.d/*
COPY docker-entrypoint.d/* /docker-entrypoint.d/docker-entrypoint.d scripts from the
standard image and adding a new one that configures the web server as we want
using a couple of environment variables:
AUTH_REQUEST_URI: URL to use for the auth_request, if the variable is not
found on the environment auth_request is not used.HTML_ROOT: Base directory of the web server, if not passed the default
/usr/share/nginx/html is used.nginx image.
The contents of the configuration script are:
#!/bin/sh
# Replace the default.conf nginx file by our own version.
set -e
if [ -z "$HTML_ROOT" ]; then
HTML_ROOT="/usr/share/nginx/html"
fi
if [ "$AUTH_REQUEST_URI" ]; then
cat >/etc/nginx/conf.d/default.conf <<EOF
server
listen 80;
server_name localhost;
location /
auth_request /.auth;
root $HTML_ROOT;
index index.html index.htm;
location /.auth
internal;
proxy_pass $AUTH_REQUEST_URI;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-URI \$request_uri;
error_page 500 502 503 504 /50x.html;
location = /50x.html
root /usr/share/nginx/html;
EOF
else
cat >/etc/nginx/conf.d/default.conf <<EOF
server
listen 80;
server_name localhost;
location /
root $HTML_ROOT;
index index.html index.htm;
error_page 500 502 503 504 /50x.html;
location = /50x.html
root /usr/share/nginx/html;
EOF
fi
# vim: ts=2:sw=2:et/sftp/data or /sftp/data/scs
folder as the root of the web published by this container and create an
Ingress object to provide access to it outside of our kubernetes cluster.webhook-scs container is generated using the following Dockerfile:
ARG ALPINE_VERSION=3.16.2
ARG GOLANG_VERSION=alpine3.16
FROM golang:$GOLANG_VERSION AS builder
LABEL maintainer="Sergio Talens-Oliag <sto@mixinet.net>"
ENV WEBHOOK_VERSION 2.8.0
ENV WEBHOOK_PR 549
ENV S3FS_VERSION v1.91
WORKDIR /go/src/github.com/adnanh/webhook
RUN apk update &&\
apk add --no-cache -t build-deps curl libc-dev gcc libgcc patch
RUN curl -L --silent -o webhook.tar.gz\
https://github.com/adnanh/webhook/archive/$ WEBHOOK_VERSION .tar.gz &&\
tar xzf webhook.tar.gz --strip 1 &&\
curl -L --silent -o $ WEBHOOK_PR .patch\
https://patch-diff.githubusercontent.com/raw/adnanh/webhook/pull/$ WEBHOOK_PR .patch &&\
patch -p1 < $ WEBHOOK_PR .patch &&\
go get -d && \
go build -o /usr/local/bin/webhook
WORKDIR /src/s3fs-fuse
RUN apk update &&\
apk add ca-certificates build-base alpine-sdk libcurl automake autoconf\
libxml2-dev libressl-dev mailcap fuse-dev curl-dev
RUN curl -L --silent -o s3fs.tar.gz\
https://github.com/s3fs-fuse/s3fs-fuse/archive/refs/tags/$S3FS_VERSION.tar.gz &&\
tar xzf s3fs.tar.gz --strip 1 &&\
./autogen.sh &&\
./configure --prefix=/usr/local &&\
make -j && \
make install
FROM alpine:$ALPINE_VERSION
LABEL maintainer="Sergio Talens-Oliag <sto@mixinet.net>"
WORKDIR /webhook
RUN apk update &&\
apk add --no-cache ca-certificates mailcap fuse libxml2 libcurl libgcc\
libstdc++ rsync util-linux-misc &&\
rm -rf /var/cache/apk/*
COPY --from=builder /usr/local/bin/webhook /usr/local/bin/webhook
COPY --from=builder /usr/local/bin/s3fs /usr/local/bin/s3fs
COPY entrypoint.sh /
COPY hooks/* ./hooks/
EXPOSE 9000
ENTRYPOINT ["/entrypoint.sh"]
CMD ["server"]PATCH included on this
pull request against a released
version of the source instead of creating a fork.
The entrypoint.sh script is used to generate the webhook configuration file
for the existing hooks using environment variables (basically the
WEBHOOK_WORKDIR and the *_TOKEN variables) and launch the webhook
service:
#!/bin/sh
set -e
# ---------
# VARIABLES
# ---------
WEBHOOK_BIN="$ WEBHOOK_BIN:-/webhook/hooks "
WEBHOOK_YML="$ WEBHOOK_YML:-/webhook/scs.yml "
WEBHOOK_OPTS="$ WEBHOOK_OPTS:--verbose "
# ---------
# FUNCTIONS
# ---------
print_du_yml()
cat <<EOF
- id: du
execute-command: '$WEBHOOK_BIN/du.sh'
command-working-directory: '$WORKDIR'
response-headers:
- name: 'Content-Type'
value: 'application/json'
http-methods: ['GET']
include-command-output-in-response: true
include-command-output-in-response-on-error: true
pass-arguments-to-command:
- source: 'url'
name: 'path'
pass-environment-to-command:
- source: 'string'
envname: 'OUTPUT_FORMAT'
name: 'json'
EOF
print_hardlink_yml()
cat <<EOF
- id: hardlink
execute-command: '$WEBHOOK_BIN/hardlink.sh'
command-working-directory: '$WORKDIR'
http-methods: ['GET']
include-command-output-in-response: true
include-command-output-in-response-on-error: true
EOF
print_s3sync_yml()
cat <<EOF
- id: s3sync
execute-command: '$WEBHOOK_BIN/s3sync.sh'
command-working-directory: '$WORKDIR'
http-methods: ['POST']
include-command-output-in-response: true
include-command-output-in-response-on-error: true
pass-environment-to-command:
- source: 'payload'
envname: 'AWS_KEY'
name: 'aws.key'
- source: 'payload'
envname: 'AWS_SECRET_KEY'
name: 'aws.secret_key'
- source: 'payload'
envname: 'S3_BUCKET'
name: 's3.bucket'
- source: 'payload'
envname: 'S3_REGION'
name: 's3.region'
- source: 'payload'
envname: 'S3_PATH'
name: 's3.path'
- source: 'payload'
envname: 'SCS_PATH'
name: 'scs.path'
stream-command-output: true
EOF
print_token_yml()
if [ "$1" ]; then
cat << EOF
trigger-rule:
match:
type: 'value'
value: '$1'
parameter:
source: 'header'
name: 'X-Webhook-Token'
EOF
fi
exec_webhook()
# Validate WORKDIR
if [ -z "$WEBHOOK_WORKDIR" ]; then
echo "Must define the WEBHOOK_WORKDIR variable!" >&2
exit 1
fi
WORKDIR="$(realpath "$WEBHOOK_WORKDIR" 2>/dev/null)" true
if [ ! -d "$WORKDIR" ]; then
echo "The WEBHOOK_WORKDIR '$WEBHOOK_WORKDIR' is not a directory!" >&2
exit 1
fi
# Get TOKENS, if the DU_TOKEN or HARDLINK_TOKEN is defined that is used, if
# not if the COMMON_TOKEN that is used and in other case no token is checked
# (that is the default)
DU_TOKEN="$ DU_TOKEN:-$COMMON_TOKEN "
HARDLINK_TOKEN="$ HARDLINK_TOKEN:-$COMMON_TOKEN "
S3_TOKEN="$ S3_TOKEN:-$COMMON_TOKEN "
# Create webhook configuration
print_du_yml
print_token_yml "$DU_TOKEN"
echo ""
print_hardlink_yml
print_token_yml "$HARDLINK_TOKEN"
echo ""
print_s3sync_yml
print_token_yml "$S3_TOKEN"
>"$WEBHOOK_YML"
# Run the webhook command
# shellcheck disable=SC2086
exec webhook -hooks "$WEBHOOK_YML" $WEBHOOK_OPTS
# ----
# MAIN
# ----
case "$1" in
"server") exec_webhook ;;
*) exec "$@" ;;
esacentrypoint.sh script generates the configuration file for the webhook
server calling functions that print a yaml section for each hook and
optionally adds rules to validate access to them comparing the value of a
X-Webhook-Token header against predefined values.
The expected token values are taken from environment variables, we can define
a token variable for each hook (DU_TOKEN, HARDLINK_TOKEN or S3_TOKEN)
and a fallback value (COMMON_TOKEN); if no token variable is defined for a
hook no check is done and everybody can call it.
The Hook
Definition documentation explains the options you can use for each hook, the
ones we have right now do the following:
du: runs on the $WORKDIR directory, passes as first argument to the
script the value of the path query parameter and sets the variable
OUTPUT_FORMAT to the fixed value json (we use that to print the output of
the script in JSON format instead of text).hardlink: runs on the $WORKDIR directory and takes no parameters.s3sync: runs on the $WORKDIR directory and sets a lot of environment
variables from values read from the JSON encoded payload sent by the caller
(all the values must be sent by the caller even if they are assigned an empty
value, if they are missing the hook fails without calling the script); we
also set the stream-command-output value to true to make the script show
its output as it is working (we patched the webhook source to be able to
use this option).du hook scriptThe du hook script code checks if the argument passed is a directory,
computes its size using the du command and prints the results in text format
or as a JSON dictionary:
#!/bin/sh
set -e
# Script to print disk usage for a PATH inside the scs folder
# ---------
# FUNCTIONS
# ---------
print_error()
if [ "$OUTPUT_FORMAT" = "json" ]; then
echo " \"error\":\"$*\" "
else
echo "$*" >&2
fi
exit 1
usage()
if [ "$OUTPUT_FORMAT" = "json" ]; then
echo " \"error\":\"Pass arguments as '?path=XXX\" "
else
echo "Usage: $(basename "$0") PATH" >&2
fi
exit 1
# ----
# MAIN
# ----
if [ "$#" -eq "0" ] [ -z "$1" ]; then
usage
fi
if [ "$1" = "." ]; then
DU_PATH="./"
else
DU_PATH="$(find . -name "$1" -mindepth 1 -maxdepth 1)" true
fi
if [ -z "$DU_PATH" ] [ ! -d "$DU_PATH/." ]; then
print_error "The provided PATH ('$1') is not a directory"
fi
# Print disk usage in bytes for the given PATH
OUTPUT="$(du -b -s "$DU_PATH")"
if [ "$OUTPUT_FORMAT" = "json" ]; then
# Format output as "path":"PATH","bytes":"BYTES"
echo "$OUTPUT"
sed -e "s%^\(.*\)\t.*/\(.*\)$% \"path\":\"\2\",\"bytes\":\"\1\" %"
tr -d '\n'
else
# Print du output as is
echo "$OUTPUT"
fi
# vim: ts=2:sw=2:et:ai:sts=2hardlink hook scriptThe hardlink hook script is really simple, it just runs the
util-linux version of the
hardlink
command on its working directory:
#!/bin/sh
hardlink --ignore-time --maximize .s3sync hook scriptThe s3sync hook script uses the s3fs
tool to mount a bucket and synchronise data between a folder inside the bucket
and a directory on the filesystem using rsync; all values needed to execute
the task are taken from environment variables:
#!/bin/ash
set -euo pipefail
set -o errexit
set -o errtrace
# Functions
finish()
ret="$1"
echo ""
echo "Script exit code: $ret"
exit "$ret"
# Check variables
if [ -z "$AWS_KEY" ] [ -z "$AWS_SECRET_KEY" ] [ -z "$S3_BUCKET" ]
[ -z "$S3_PATH" ] [ -z "$SCS_PATH" ]; then
[ "$AWS_KEY" ] echo "Set the AWS_KEY environment variable"
[ "$AWS_SECRET_KEY" ] echo "Set the AWS_SECRET_KEY environment variable"
[ "$S3_BUCKET" ] echo "Set the S3_BUCKET environment variable"
[ "$S3_PATH" ] echo "Set the S3_PATH environment variable"
[ "$SCS_PATH" ] echo "Set the SCS_PATH environment variable"
finish 1
fi
if [ "$S3_REGION" ] && [ "$S3_REGION" != "us-east-1" ]; then
EP_URL="endpoint=$S3_REGION,url=https://s3.$S3_REGION.amazonaws.com"
else
EP_URL="endpoint=us-east-1"
fi
# Prepare working directory
WORK_DIR="$(mktemp -p "$HOME" -d)"
MNT_POINT="$WORK_DIR/s3data"
PASSWD_S3FS="$WORK_DIR/.passwd-s3fs"
# Check the moutpoint
if [ ! -d "$MNT_POINT" ]; then
mkdir -p "$MNT_POINT"
elif mountpoint "$MNT_POINT"; then
echo "There is already something mounted on '$MNT_POINT', aborting!"
finish 1
fi
# Create password file
touch "$PASSWD_S3FS"
chmod 0400 "$PASSWD_S3FS"
echo "$AWS_KEY:$AWS_SECRET_KEY" >"$PASSWD_S3FS"
# Mount s3 bucket as a filesystem
s3fs -o dbglevel=info,retries=5 -o "$EP_URL" -o "passwd_file=$PASSWD_S3FS" \
"$S3_BUCKET" "$MNT_POINT"
echo "Mounted bucket '$S3_BUCKET' on '$MNT_POINT'"
# Remove the password file, just in case
rm -f "$PASSWD_S3FS"
# Check source PATH
ret="0"
SRC_PATH="$MNT_POINT/$S3_PATH"
if [ ! -d "$SRC_PATH" ]; then
echo "The S3_PATH '$S3_PATH' can't be found!"
ret=1
fi
# Compute SCS_UID & SCS_GID (by default based on the working directory owner)
SCS_UID="$ SCS_UID:=$(stat -c "%u" "." 2>/dev/null) " true
SCS_GID="$ SCS_GID:=$(stat -c "%g" "." 2>/dev/null) " true
# Check destination PATH
DST_PATH="./$SCS_PATH"
if [ "$ret" -eq "0" ] && [ -d "$DST_PATH" ]; then
mkdir -p "$DST_PATH" ret="$?"
fi
# Copy using rsync
if [ "$ret" -eq "0" ]; then
rsync -rlptv --chown="$SCS_UID:$SCS_GID" --delete --stats \
"$SRC_PATH/" "$DST_PATH/" ret="$?"
fi
# Unmount the S3 bucket
umount -f "$MNT_POINT"
echo "Called umount for '$MNT_POINT'"
# Remove mount point dir
rmdir "$MNT_POINT"
# Remove WORK_DIR
rmdir "$WORK_DIR"
# We are done
finish "$ret"
# vim: ts=2:sw=2:et:ai:sts=2StatefulSet with one replica.
Our production deployment is done on AWS and to be
able to scale we use EFS for our
PersistenVolume; the idea is that the volume has no size limit, its
AccessMode can be set to ReadWriteMany and we can mount it from multiple
instances of the Pod without issues, even if they are in different availability
zones.
For development we use k3d and we are also able to scale the
StatefulSet for testing because we use a ReadWriteOnce PVC, but it points
to a hostPath that is backed up by a folder that is mounted on all the
compute nodes, so in reality Pods in different k3d nodes use the same folder
on the host.
mysecureshell container that
can be generated using kubernetes pods as follows (we are only creating the
scs user):
$ kubectl run "mysecureshell" --restart='Never' --quiet --rm --stdin \
--image "stodh/mysecureshell:latest" -- gen-host-keys >"./host_keys.txt"
$ kubectl run "mysecureshell" --restart='Never' --quiet --rm --stdin \
--image "stodh/mysecureshell:latest" -- gen-users-tar scs >"./users.tar"secrets.yaml file as follows:
$ tar xf ./users.tar user_keys.txt user_pass.txt
$ kubectl --dry-run=client -o yaml create secret generic "scs-secret" \
--from-file="host_keys.txt=host_keys.txt" \
--from-file="user_keys.txt=user_keys.txt" \
--from-file="user_pass.txt=user_pass.txt" > ./secrets.yamlsecrets.yaml will look like the following file (the base64
would match the content of the files, of course):
apiVersion: v1
data:
host_keys.txt: TWlt...
user_keys.txt: c2Nz...
user_pass.txt: c2Nz...
kind: Secret
metadata:
creationTimestamp: null
name: scs-secretstatefulSet) can be as simple as this:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: scs-pvc
labels:
app.kubernetes.io/name: scs
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8GistorageClassName to use the default one.
PersistentVolume as
required by the
Local
Persistence Volume Static Provisioner (note that the /volumes/scs-pv has to
be created by hand, in our k3d system we mount the same host directory on the
/volumes path of all the nodes and create the scs-pv directory by hand
before deploying the persistent volume):
apiVersion: v1
kind: PersistentVolume
metadata:
name: scs-pv
labels:
app.kubernetes.io/name: scs
spec:
capacity:
storage: 8Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
claimRef:
name: scs-pvc
storageClassName: local-storage
local:
path: /volumes/scs-pv
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: node.kubernetes.io/instance-type
operator: In
values:
- k3sstorageClassName:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: scs-pvc
labels:
app.kubernetes.io/name: scs
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: local-storagePersistentVolume (we are
using the
aws-efs-csi-driver which
supports Dynamic Provisioning) but we add the storageClassName (we set it
to the one mapped to the EFS driver, i.e. efs-sc) and set ReadWriteMany
as the accessMode:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: scs-pvc
labels:
app.kubernetes.io/name: scs
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 8Gi
storageClassName: efs-scstatefulSet is as follows:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: scs
labels:
app.kubernetes.io/name: scs
spec:
serviceName: scs
replicas: 1
selector:
matchLabels:
app: scs
template:
metadata:
labels:
app: scs
spec:
containers:
- name: nginx
image: stodh/nginx-scs:latest
ports:
- containerPort: 80
name: http
env:
- name: AUTH_REQUEST_URI
value: ""
- name: HTML_ROOT
value: /sftp/data
volumeMounts:
- mountPath: /sftp
name: scs-datadir
- name: mysecureshell
image: stodh/mysecureshell:latest
ports:
- containerPort: 22
name: ssh
securityContext:
capabilities:
add:
- IPC_OWNER
env:
- name: SFTP_UID
value: '2020'
- name: SFTP_GID
value: '2020'
volumeMounts:
- mountPath: /secrets
name: scs-file-secrets
readOnly: true
- mountPath: /sftp
name: scs-datadir
- name: webhook
image: stodh/webhook-scs:latest
securityContext:
privileged: true
ports:
- containerPort: 9000
name: webhook-http
env:
- name: WEBHOOK_WORKDIR
value: /sftp/data/scs
volumeMounts:
- name: devfuse
mountPath: /dev/fuse
- mountPath: /sftp
name: scs-datadir
volumes:
- name: devfuse
hostPath:
path: /dev/fuse
- name: scs-file-secrets
secret:
secretName: scs-secrets
- name: scs-datadir
persistentVolumeClaim:
claimName: scs-pvcnginx: As this is an example the web server is not using an
AUTH_REQUEST_URI and uses the /sftp/data directory as the root of the web
(to get to the files uploaded for the scs user we will need to use /scs/
as a prefix on the URLs).mysecureshell: We are adding the IPC_OWNER capability to the container to
be able to use some of the sftp-* commands inside it, but they are
not really needed, so adding the capability is optional.webhook: We are launching this container in privileged mode to be able to
use the s3fs-fuse, as it will not work otherwise for now (see this
kubernetes issue); if
the functionality is not needed the container can be executed with regular
privileges; besides, as we are not enabling public access to this service we
don t define *_TOKEN variables (if required the values should be read from a
Secret object).devfuse volume is only needed if we plan to use the s3fs command on
the webhook container, if not we can remove the volume definition and its
mounts.Service object:
apiVersion: v1
kind: Service
metadata:
name: scs-svc
labels:
app.kubernetes.io/name: scs
spec:
ports:
- name: ssh
port: 22
protocol: TCP
targetPort: 22
- name: http
port: 80
protocol: TCP
targetPort: 80
- name: webhook-http
port: 9000
protocol: TCP
targetPort: 9000
selector:
app: scsscs files from the outside we can add an ingress object like
the following (the definition is for testing using the localhost name):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: scs-ingress
labels:
app.kubernetes.io/name: scs
spec:
ingressClassName: nginx
rules:
- host: 'localhost'
http:
paths:
- path: /scs
pathType: Prefix
backend:
service:
name: scs-svc
port:
number: 80statefulSet we create a namespace and apply the object
definitions shown before:
$ kubectl create namespace scs-demo
namespace/scs-demo created
$ kubectl -n scs-demo apply -f secrets.yaml
secret/scs-secrets created
$ kubectl -n scs-demo apply -f pvc.yaml
persistentvolumeclaim/scs-pvc created
$ kubectl -n scs-demo apply -f statefulset.yaml
statefulset.apps/scs created
$ kubectl -n scs-demo apply -f service.yaml
service/scs-svc created
$ kubectl -n scs-demo apply -f ingress.yaml
ingress.networking.k8s.io/scs-ingress createdkubectl:
$ kubectl -n scs-demo get all,secrets,ingress
NAME READY STATUS RESTARTS AGE
pod/scs-0 3/3 Running 0 24s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/scs-svc ClusterIP 10.43.0.47 <none> 22/TCP,80/TCP,9000/TCP 21s
NAME READY AGE
statefulset.apps/scs 1/1 24s
NAME TYPE DATA AGE
secret/default-token-mwcd7 kubernetes.io/service-account-token 3 53s
secret/scs-secrets Opaque 3 39s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/scs-ingress nginx localhost 172.21.0.5 80 17ssftp server from
other Pods, but to test the system we are going to do a kubectl port-forward
and connect to the server using our host client and the password we have
generated (it is on the user_pass.txt file, inside the users.tar archive):
$ kubectl -n scs-demo port-forward service/scs-svc 2020:22 &
Forwarding from 127.0.0.1:2020 -> 22
Forwarding from [::1]:2020 -> 22
$ PF_PID=$!
$ sftp -P 2020 scs@127.0.0.1 1
Handling connection for 2020
The authenticity of host '[127.0.0.1]:2020 ([127.0.0.1]:2020)' can't be \
established.
ED25519 key fingerprint is SHA256:eHNwCnyLcSSuVXXiLKeGraw0FT/4Bb/yjfqTstt+088.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[127.0.0.1]:2020' (ED25519) to the list of known \
hosts.
scs@127.0.0.1's password: **********
Connected to 127.0.0.1.
sftp> ls -la
drwxr-xr-x 2 sftp sftp 4096 Sep 25 14:47 .
dr-xr-xr-x 3 sftp sftp 4096 Sep 25 14:36 ..
sftp> !date -R > /tmp/date.txt 2
sftp> put /tmp/date.txt .
Uploading /tmp/date.txt to /date.txt
date.txt 100% 32 27.8KB/s 00:00
sftp> ls -l
-rw-r--r-- 1 sftp sftp 32 Sep 25 15:21 date.txt
sftp> ln date.txt date.txt.1 3
sftp> ls -l
-rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt
-rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt.1
sftp> put /tmp/date.txt date.txt.2 4
Uploading /tmp/date.txt to /date.txt.2
date.txt 100% 32 27.8KB/s 00:00
sftp> ls -l 5
-rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt
-rw-r--r-- 2 sftp sftp 32 Sep 25 15:21 date.txt.1
-rw-r--r-- 1 sftp sftp 32 Sep 25 15:21 date.txt.2
sftp> exit
$ kill "$PF_PID"
[1] + terminated kubectl -n scs-demo port-forward service/scs-svc 2020:22sftp service on the forwarded port with the scs user.date.txt file from the
URL http://localhost/scs/date.txt:
$ curl -s http://localhost/scs/date.txt
Sun, 25 Sep 2022 17:21:51 +0200hooks directly,
from a CronJob and from a Job.
du)In our deployment the direct calls are done from other Pods, to simulate it we
are going to do a port-forward and call the script with an existing PATH (the
root directory) and a bad one:
$ kubectl -n scs-demo port-forward service/scs-svc 9000:9000 >/dev/null &
$ PF_PID=$!
$ JSON="$(curl -s "http://localhost:9000/hooks/du?path=.")"
$ echo $JSON
"path":"","bytes":"4160"
$ JSON="$(curl -s "http://localhost:9000/hooks/du?path=foo")"
$ echo $JSON
"error":"The provided PATH ('foo') is not a directory"
$ kill $PF_PID.
PATH and the output is in json format because we export OUTPUT_FORMAT with
the value json on the webhook configuration.hardlink)As explained before, the webhook container can be used to run cronjobs; the
following one uses an alpine container to call the hardlink script each
minute (that setup is for testing, obviously):
apiVersion: batch/v1
kind: CronJob
metadata:
name: hardlink
labels:
cronjob: 'hardlink'
spec:
schedule: "* */1 * * *"
concurrencyPolicy: Replace
jobTemplate:
spec:
template:
metadata:
labels:
cronjob: 'hardlink'
spec:
containers:
- name: hardlink-cronjob
image: alpine:latest
command: ["wget", "-q", "-O-", "http://scs-svc:9000/hooks/hardlink"]
restartPolicy: Never$ kubectl -n scs-demo apply -f webhook-cronjob.yaml 1
cronjob.batch/hardlink created
$ kubectl -n scs-demo get pods -l "cronjob=hardlink" -w 2
NAME READY STATUS RESTARTS AGE
hardlink-27735351-zvpnb 0/1 Pending 0 0s
hardlink-27735351-zvpnb 0/1 ContainerCreating 0 0s
hardlink-27735351-zvpnb 0/1 Completed 0 2s
^C
$ kubectl -n scs-demo logs pod/hardlink-27735351-zvpnb 3
Mode: real
Method: sha256
Files: 3
Linked: 1 files
Compared: 0 xattrs
Compared: 1 files
Saved: 32 B
Duration: 0.000220 seconds
$ sleep 60
$ kubectl -n scs-demo get pods -l "cronjob=hardlink" 4
NAME READY STATUS RESTARTS AGE
hardlink-27735351-zvpnb 0/1 Completed 0 83s
hardlink-27735352-br5rn 0/1 Completed 0 23s
$ kubectl -n scs-demo logs pod/hardlink-27735352-br5rn 5
Mode: real
Method: sha256
Files: 3
Linked: 0 files
Compared: 0 xattrs
Compared: 0 files
Saved: 0 B
Duration: 0.000070 seconds
$ kubectl -n scs-demo delete -f webhook-cronjob.yaml 6
cronjob.batch "hardlink" deletedcronjob label, we interrupt it once we see
that the first run has been completed.date.txt.2 has been replaced by a hardlink (the
summary does not name the file, but it is the only option knowing the
contents from the original upload).s3sync)The following job can be used to synchronise the contents of a directory in a
S3 bucket with the SCS Filesystem:
apiVersion: batch/v1
kind: Job
metadata:
name: s3sync
labels:
cronjob: 's3sync'
spec:
template:
metadata:
labels:
cronjob: 's3sync'
spec:
containers:
- name: s3sync-job
image: alpine:latest
command:
- "wget"
- "-q"
- "--header"
- "Content-Type: application/json"
- "--post-file"
- "/secrets/s3sync.json"
- "-O-"
- "http://scs-svc:9000/hooks/s3sync"
volumeMounts:
- mountPath: /secrets
name: job-secrets
readOnly: true
restartPolicy: Never
volumes:
- name: job-secrets
secret:
secretName: webhook-job-secrets
"aws":
"key": "********************",
"secret_key": "****************************************"
,
"s3":
"region": "eu-north-1",
"bucket": "blogops-test",
"path": "test"
,
"scs":
"path": "test"
$ kubectl -n scs-demo create secret generic webhook-job-secrets \ 1
--from-file="s3sync.json=s3sync.json"
secret/webhook-job-secrets created
$ kubectl -n scs-demo apply -f webhook-job.yaml 2
job.batch/s3sync created
$ kubectl -n scs-demo get pods -l "cronjob=s3sync" 3
NAME READY STATUS RESTARTS AGE
s3sync-zx2cj 0/1 Completed 0 12s
$ kubectl -n scs-demo logs s3sync-zx2cj 4
Mounted bucket 's3fs-test' on '/root/tmp.jiOjaF/s3data'
sending incremental file list
created directory ./test
./
kyso.png
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 2 (reg: 1, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 15,075 bytes
Total transferred file size: 15,075 bytes
Literal data: 15,075 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.147 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 15,183
Total bytes received: 74
sent 15,183 bytes received 74 bytes 30,514.00 bytes/sec
total size is 15,075 speedup is 0.99
Called umount for '/root/tmp.jiOjaF/s3data'
Script exit code: 0
$ kubectl -n scs-demo delete -f webhook-job.yaml 5
job.batch "s3sync" deleted
$ kubectl -n scs-demo delete secrets webhook-job-secrets 6
secret "webhook-job-secrets" deletedwebhook-job-secrets secret that contains the
s3sync.json file.cronjob=s3sync we get the Pods executed by the job.
Welcome to the August 2022 report from the Reproducible Builds project! In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.
As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.
The US National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA) and the Office of the Director of National Intelligence (ODNI) have released a document called Securing the Software Supply Chain: Recommended Practices Guide for Developers (PDF) as part of their Enduring Security Framework (ESF) work.
The document expressly recommends having reproducible builds as part of advanced recommended mitigations, along with hermetic builds. Page 31 (page 35 in the PDF) says:
Reproducible builds provide additional protection and validation against attempts to compromise build systems. They ensure the binary products of each build system match: i.e., they are built from the same source, regardless of variable metadata such as the order of input files, timestamps, locales, and paths. Reproducible builds are those where re-running the build steps with identical input artifacts results in bit-for-bit identical output. Builds that cannot meet this must provide a justification why the build cannot be made reproducible.The full press release is available online.
On our mailing list this month, Marc Prud hommeaux posted a feature request for diffoscope which additionally outlines a project called The App Fair, an autonomous distribution network of free and open-source macOS and iOS applications, where validated apps are then signed and submitted for publication .
This is the core of a two-decade-old debate among security people, and it s one that the benevolent God faction has consistently had the upper hand in. They re the curated computing advocates who insist that preventing you from choosing an alternative app store or side-loading a program is for your own good because if it s possible for you to override the manufacturer s wishes, then malicious software may impersonate you to do so, or you might be tricked into doing so. [..] This benevolent dictatorship model only works so long as the dictator is both perfectly benevolent and perfectly competent. We know the dictators aren t always benevolent. [ ] But even if you trust a dictator s benevolence, you can t trust in their perfection. Everyone makes mistakes. Benevolent dictator computing works well, but fails badly. Designing a computer that intentionally can t be fully controlled by its owner is a nightmare, because that is a computer that, once compromised, can attack its owner with impunity.
In Debian this month, the essential and required package sets became 100% reproducible in Debian bookworm on the amd64 and arm64 architectures. These two subsets of the full Debian archive refer to Debian package priority levels as described in the 2.5 Priorities section of the Debian Policy there is no canonical minimal installation package set in Debian due to its diverse methods of installation.
As it happens, these package sets are not reproducible on the i386 architecture because the ncurses package on that architecture is not yet reproducible, and the sed package currently fails to build from source on armhf too. The full list of reproducible packages within these package sets can be viewed within our QA system, such as on the page of required packages in amd64 and the list of essential packages on arm64, both for Debian bullseye.
podman on Debian bullseye:
$ sudo apt install podman
$ podman run --rm -it debian:bullseye bash
$ SOURCE_DATE_EPOCH=$(date --utc --date=2022-08-29 +%s) mmdebstrap unstable > unstable.tar
$SOURCE_DATE_EPOCH to be not greater than SOURCE_DATE_EPOCH.
/etc/machine-id, /var/cache/ldconfig/aux-cache, /var/log/dpkg.log, /var/log/alternatives.log and /var/log/bootstrap.log, and for cdebootstrap we also need to delete the /var/log/apt/history.log and /var/log/apt/term.log files as well.
/etc/machine-id file in both debootstrap [ ] and cdebootstrap [ ].
randomness_in_browserify_output [ ], haskell_abi_hash_differences [ ], nondeterministic_ids_in_html_output_generated_by_python_sphinx_panels [ ]. Lastly, Mattia Rizzolo removed the deterministic flag from the captures_kernel_variant flag [ ].
Vagrant Cascadian posted an update of the status of Reproducible Builds in GNU Guix, writing that:
The post itself contains a lot more details, including a brief discussion of tooling. Elsewhere in GNU Guix, however, Vagrant updated a number of packages such asIgnoring the pesky unknown packages, it is more like ~93% reproducibleand ~7% unreproducible... that feels a bit better to me!These numbers wander around over time, mostly due to packages movingback into an "unknown" state while the build farms catch up with eachother... although the above numbers seem to have been pretty consistentover the last few days.
itpp [ ], perl-class-methodmaker [ ], libnet [ ], directfb [ ] and mm-common [ ], as well as updated the version of reprotest to 0.7.21 [ ].
In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.
diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 220 and 221 to Debian, as well as made the following changes:
external_tools.py to reflect changes to xxd and the vim-common package. [ ]xxd package now, not the vim-common package. [ ]at-spi-sharp (build failure when build on a multiprocessor machine).borgbackup (fails to build in 2038, fix)buzztrax (parallelism-related issue)chart-testing (date-related issue)memcached (fails to build in 2038)nim (fails to build in 2038)perl-Time-Moment (fails to build in 2038)python-bson (fails to build in 2038)python-heatclient (fails to build in 2038)python3.8 (fails to build in 2038)reproducible-faketoolss3fs (date-related issue)systemd (date-related issue)wayfire.multipath-tools.node-canvas-confetti.psi (forwarded upstream).sphinx-panels (forwarded upstream).sysfsutils.geeqie.
The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, Holger Levsen made the following changes:
deb-src lines to enable test builds a Non-maintainer Upload (NMU) campaign targeting 708 sources without .buildinfo files found in Debian unstable, including 475 in bookworm. [ ][ ]linux-image-generic kernel package installed. [ ]SOURCE_DATE_EPOCH for all our new bootstrap jobs. [ ]/bin/sh symlink [ ].
#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
I arrived in Prizren late on Wednesday. Here s what I did during debcamp (so over 3 days). I hope this post just motivates others to contribute more to Debian.
At least 2 DDs want to upload packages that need a new version of python3-jsonschema (ie: version > 4.x). Unfortunately, version 4 broke a few packages. I therefore uploaded it to Experimental a few months/week, so I could see the result of autopkgtest reading the pseudo excuse page. And it showed a few packages broke. Here s the one used (or part of) OpenStack:
Engineers build systems. Good engineers always stress and focus efficiency of these systems.
Two recent examples of engineering thinking follow. One was in a video / podcast interview with Martin Thompson (who is a noted high-performance code expert) I came across recently. The overall focus of the hour-long interview is on managing software complexity . Around minute twenty-two, the conversation turns to feedback loops and systems, and a strong preference for simple and fast systems for more immediate feedback. An important topic indeed.
The second example connects to this and permeates many tweets and other writings by Erik Bernhardsson. He had an earlier 2017 post on Optimizing for iteration speed , as well as a 17 May 2022 tweet on minimizing feedback loop size, another 28 Mar 2022 tweet reply on shorter feedback loops, then a 14 Feb 2022 post on problems with slow feedback loops, as well as a 13 Jan 2022 post on a priority for tighter feedback loops, and lastly a 23 Jul 2021 post on fast feedback cycles. You get the idea: Erik really digs faster feedback loops. Nobody likes to wait: immediatecy wins each time.
A few years ago, I had touched on this topic with two posts on how to make (R) package compilation (and hence installation) faster. One idea (which I still use whenever I must compile) was in post #11 on caching compilation. Another idea was in post #13: make it faster by not doing it, in this case via binary installation which skip the need for compilation (and which is what I aim for with, say, CI dependencies). Several subsequent posts can be found by scrolling down the r^4 blog section: we stressed the use of the amazing Rutter PPA c2d4u for CRAN binaries (often via Rocker containers, the (post #28) promise of RSPM, and the (post #29) awesomeness of bspm. And then in the more recent post #34 from last December we got back to a topic which ties all these things together: Dependencies. We quoted Mies van der Rohe: Less is more. Especially when it comes to dependencies as these elongate the feedback loop and thereby delay feedback.
Our most recent post #37 on r2u connects these dots. Access to a complete set of CRAN binaries with full-dependency resolution accelerates use and installation. This of course also covers testing and continuous integration. Why wait minutes to recompile the same packages over and over when you can install the full Tidyverse in 18 seconds or the brms package and all it needs in 13 seconds as shown in the two gifs also on the r2u documentation site.
You can even power up the example setup of the second gif via this gitpod link giving you a full Ubuntu 22.04 session in your browser to try this: so go forth and install something from CRAN with ease! The benefit of a system such our r2u CRAN binaries is clear: faster feedback loops. This holds whether you work with few or many dependencies, tiny or tidy. Faster matters, and feedback can be had sooner.
And with the title of this post we now get a rallying cry to advocate for faster feedback systems: FFS .
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
The following contributors got their Debian Developer accounts in the last two months:
Next.
